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TITLE OF THE INVENTION 

A PEN-MOUSE SYSTEM 
CROSS-REFERENCE TO RELATED APPLICATION(S) 

[0001] This application is related to and claims priority to U.S. provisional application 
entitled Tracking Menu System (1252.1082P) having serial number 60/419,144, by Fitzmaurice 
et al filed October 18, 2002, this application is also related to U.S. application entitled Tracking 

Menu System and Method having serial number (1252.1082), by Fitzmaurice et al. filed 

concurrently herewith and to U.S. application entitled Pan-Zoom Tool having serial number 

(1252.1086), by Fitzmaurice et al, filed concurrently herewith, all of which are incorporated by 
reference herein. 

BACKGROUND OF THE INVENTION 

1. Field of the Invention 

[0002] The present invention is directed to a tracking menu that can be used with a pen 
based computer and that provides a pen user the functionality of a mouse and, more 
particularly, to a pen-mouse that tracks a position of a pen or stylus, allows pointing like a 
mouse and emulates the functionality of mouse buttons and other mouse inputs, such as a 
finger reliable wheel. 

2. Description of the Related Art 

[0003] Stylus or pen-based computers, such as tablet (personal computers) PCs and 
personal digital assistants (PDAs) are becoming a popular type of computing device. These 
devices operate somewhat differently than the typical mouse-based computer systems. In the 
pen-based devices the user uses a stylus or pen to select and activate items, such as menu 
controls/buttons, as well as to perform graphic functions, such as drawing. In performing these 
different operations it is often the case that the user must move the pen to a menu to select a 
function and then return to the display area to perform the function. Because these pen-based 
operations are somewhat different from traditional mouse based operations, it can be helpful to 
a new user to have a familiar paradigm, such as a mouse, that can be used in pen-based 
computers to perform mouse type operations. 

[0004] What is needed is an interface that provides a user with the look and "feel" of 
operation of a mouse and avoids the need to move to a menu to select mouse functions, such 
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as left button, right button, scroll, etc. 
SUMMARY OF THE INVENTION 

[0005] It is an aspect of the present invention to provide a graphical user interface that 
emulates a mouse input device and is controlled by a pen of a pen based computer. 
l0 006] It is another aspect of the present invention to provide a mouse emulation that 
functions as a tracking menu. 

[0007] It is also an aspect of the present invention is to provide mouse emulation that does 
not use keyboard events/signals but does emulate some keyboard keys (e.g., shift/alt) 
[0008] The above aspects can be attained by a system that provides a pen based computer 
user with a tool, a pen-mouse, that functions like a mouse but that is controlled by a limited input 
device, such as a pen or stylus. The pen-mouse is a tracking menu that tracks the posit.cn of 
the pen A pen-cursor that corresponds to the pen is allowed to move about within a pen- 
mouse graphic. The pen-mouse is moved when the location of the pen, pen cursor or pen 
transducer sensed position encounters a tracking boundary of the pen-mouse. While movmg 
within the pen-mouse the pen can select objects within the pen-mouse such as buttons, wheels, 
etc The selection of a button or other virtual control causes a corresponding mouse button 
function to be executed. The execution focus is directed at any object designated by a track.ng 
symbol, such as an arrow, that is part of the pen-mouse graphic. 
[0009] These together with other aspects and advantages which will be subsequently 
apparent, reside in the details of construction and operation as more fully hereinafter descnbed 
and claimed, reference being had to the accompanying drawings forming a part hereof, where.n 
like numerals refer to like parts throughout. 
BRIEF DESCRIPTION OF THE DRAWINGS 

[0010] Figure 1 illustrates a pen-mouse tracking menu according to the present invention. 

[001 1] Figure 2 depicts a tracking border of the pen-mouse. 

[0012] Figure 3 illustrates tracking states. 

[0013] Figure 4 shows a pen cursor moving within the pen-mouse. 

[0014] Figure 5 shows the pen cursor moving the pen-mouse. 

[0015] Figures 6 and 7 show a button being highlighted and selected. 
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[001 6] Figures 8-1 0 depict the pen-mouse during pan tracking. 

[0017] Figures 11 and 12 show the pen-mouse bar function highlighted and activated. 

[0018] Figures 13-23 show marking menu activation and double button function activation 

[0019] Figure 24 shows object dragging with the pen-mouse. 

[0020] Figures 25 and 26 depict linear menu activation and selection. 

[0021] Figure 27 illustrates a tracking boundary not coinciding with the body graphic. 

[0022] Figures 28a-28g illustrate different positions for the pen-mouse tracking symbol. 

[0023] Figure 29 depicts a different pen-mouse graphic with a wheel function. 

[0024] Figure 30 shows a limited function and a limited graphic pen-mouse. 

[0025] Figure 31 shows a pen-mouse with activation safety features. 

[0026] Figure 32 shows an arrow shaped pen-mouse. 

[0027] Figure 33 shows pen-mouse tracking symbols changing according to system state. 
[0028] Figure 34 shows an interior tracking wall. 

[0029] Figure 35 shows a wall being used to provide track ball functionality. 
[0030] Figure 36 shows pen-mouse relative size. 

[0031] Figure 37 shows a desktop PC as hardware of the present invention. 
[0032] Figure 38 depicts a tablet PC as hardware of the present invention. 
[0033] Figure 39 shows components of the hardware of the present invention. 
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 
[0034] The present invention is directed to a user interface, particularly, a graphical user 
interface (GUI) that has the appearance of a mouse 10 as depicted in figure 1 . This interface 
10 which for convenience will be called a pen-mouse, is a tracking menu where the pen-mouse 
lo'tracks the position of a stylus or pen being used with a pen-based computer. A track.ng 
menu is a graphical user interface widget that is controlled by either a pen or mouse havmg an 
associated pen or mouse cursor where the cursors can be invisible. A tracking menu is invoked 
and dismissed in the same manner as a traditional modal tool by clicking on a tool palette or 
menu item Like traditional menus, a tracking menu includes a cluster of graphical buttons or 
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controls The pen cursor can be moved within the menu to select and interact with items, such 
as the buttons. However, unlike Iraditonal menus, when the pen cursor crosses the extenor 
edge or tracing boundary of the menu, the menu is moved to keep it under the cursor. The 
pen-mouse tracking menu, of the present Invemion, can track other types of input dev,ces. such 
as a finger in a touch based menu, a traditional mouse and 3D input devices such as gloves. 
Additional details with respect to tracking menus can be found in the related application noted 
above. 

,0035, The pen-mouse 10 preferably includes a mouse graphic 12 that includes a visible 
border or edge 14 and a traditional arrow cursor or tracking symbol 16. This pen-mouse 
tracking menu 10 tracks the position of the pen of me pen based computer system as Will be 
discussed later in more detail. By moving the pen, the cursor 16 can be positioned to port a. 
an object in the same way that a cursor for a mouse is positioned to point at an object The 
mouse graphic 12 depicts three buttons 18 (left - L). 20 (middle - M) and 22 (right - R) that are 
conventional type virtual controls specially for initiating mouse button down events when 
activated by a pen down event with the pen tip positioned over one of the buttons 18-22. The 
.racking symbol or cursor 16 behaves like a traditional tracking symbol used with a mouse 
vrhere the Iradking symbol can point at (be positioned over) an object, such as a menu, and 
activate .he object when a mouse down even, occurs. That is. conventional input mapping 
based on location of an input sensor or cursor are performed. As a result, when the cursor 16 ,s 
positioned over an ac.iva.able object and a pen down even, occurs with the pen located over 
one of the buttons 18-22, a mouse down event for ttie corresponding mouse button occurs for 
me object a. which the cursor points. In this way, the pen of .he pen-based computer system 
can be used to perform ttie same functions as a mouse in a mouse based computer sys.em. 
,00361 The pen-mouse tracking menu 10. as depicted in figura 2, includes a tracing menu 
.racking boundary 30 that a,.ows the pen cursor 16 ,o move within .he tracking menu 10 wrtttou 
causing the menu .o pen-mouse 10 .o move with the movement of the pen. The boundary 30 ,s 
shown as coinciding with the visible edge of .he graphic 12 but need no. be so coincden. (see 
figure 27). 

.00371 The operation of the pen-mouse tracking menu 10 of the present invention can be 
understood using a state diagram as depicted in figure 3. In state 0 the pen is out-of-range 40 
of .he .racking system of .he pen based computer, such as a tabte. PC, and ttie Mq menu 
,0 is visible (preferably semi-transparent) and is positioned in the las. place ,t res,ded before the 
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pen moved out-of-range or in an initial position if the tracking menu has just been involved. The 
state 0 is entered from states 1 and 1 E when the pen is lifted from the tablet and moved out-of- 
range. The menu 10 resides in this last position until the pen is brought back into range and the 
tablet PC begins tracking the pen. 

[0038] The pen can move into tracking range 42 and enter state 1 in two different ways. In 
the first into-range transition, the pen comes down at a position where the tip of the pen .s 
outside of the tracking menu edge or tracking boundary 30. When tracking starts in th.s 
condition, the menu 10 is moved or repositioned 44 so that the cursor is inside the menu edge. 
This involves conventionally redrawing the semi-transparent menu 10 at a portion 
corresponding to the pen tip, cursor or sensed pen transducer position. This repositioning or 
redrawing can place the menu 10 such that the menu 10 is moved the minimum distance to 
place the pen cursor just within the edge of the menu 1 0. Or the repositioning can place the 
menu at an arbitrary position under the cursor, such as positioning the menu with the pen t,p „ 
the center of the menu. The second into-range transition occurs when the pen tip comes into 
range when it is within the boundary 30 of the tracking menu 10. 

[0039] in state 1 the pen cursor moves freely about within the menu 10 and the menu 10 
stays stationary. During this movement of the pen cursor within the menu 10, the system 
performs conventional operations, such as highlighting buttons or controls over which the cursor 
passes by comparing the position of the cursor to positions of the buttons. However, if the pen 
is lifted out-of-range the state moves to state 0 (the pen tracking becomes inactive), if the pen 
encounters an edge as it moves, state 1 E is entered and if the pen touches the tablet surface 
state 2 is entered. 

,00401 To enter state 1 E the position of the pen cursor (or pen tip transduoer position) is 
conventionally compared to the position of the edges or boundary 30 of the tracking menu 10. 
When a match occurs, the cursor has hit the edge and the state 1E is entered. In state 1E, as 
long as the cursor is at or in contact with an edge as the cursor moves, the tracking menu 
(semi-transparent) is moved along with the cursor. That is, as the cursor is moved, the menu 10 
is conventionally redrawn with the cursor at the edge of the tracking menu. In state 1E, * the 
pen is lifted out-of-range the stale moves to state 0, if the pen moves away from an edge to 
reenter the interior of the menu the state moves to state 1 and if the pen touches the tablet, 
state 2 is entered. 

[0041] As discussed above, state 2 is entered when the pen touches 46 the tablet surface 
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while in state 1 or state 1 E. In this state the pen is active or activated such that it will cause 
some function to be performed. In state 2 the active pen can be selecting a button, in which 
case the function of the button is performed, such as selection of a new. The tracking menu 
does not have to become transparent while in state 2. The controls or buttons of the present 
invention can be designed to work such that when you press on them, they show some 
highlighting and the assigned function is executed only on pen-up. This allows the user to 
cancel their action by moving off of the button while in the pressed state and thus preventing the 
triggering of the function. Or the active pen can be moving while under the control of a 
previously selected function, such as painting with a previously selected paintbrush or zooming 
based on a previous selection of a zoom tool/function. In state 2, the tracking menu 10 is made 
fully transparent. In this state, the system can continue to reposition the fully transparent menu 
under the cursor or preferably the menu can be allowed to remain in it's last position as in state 
0 (note the user would not perceive a difference between these two alternatives). When the pen 
is lifted from the tablet surface and contact with the tablet ends, the tracking mode 42 is again 
entered and the menu 10 is repositioned 48 depending on the last state. If the pen is lifted 
when the prior state is state 1 E, the pen is repositioned 48 at the last edge point of state 1 E. If 
the pen is lifted when the prior state is state 1 , the pen is repositioned 48 at the last interior point 
of state 1 . 

[0042] Additional details concerning tracking operations can be found in the related 
application previously noted. 

[0043] Figure 4 illustrates a pen cursor or pen transducer location symbol 40 moving to the 
left within the tracking boundary 30 of the pen-mouse 10 and the pen-mouse 10 remaining 
stationary. That is, within the body of the virtual mouse 10, movement of the sensed position 40 
of the input transducer does not cause the tracking menu 10 to move. In figure 5 the moving 
cursor 40 has encountered or hit the boundary 30 when moving to the left and the encounter 
with the boundary causes the pen-mouse 10 to move to the left. That is, movement of the 
tracking menu 10 occurs when the sensed position of input transducer 40 hits the tracking menu 
border 30 while the input transducer is in a "tracking state" (i.e., not during a mouse down state). 
[0044] As depicted in figure 6, moving the input transducer 40 over the left mouse button 
graphic 1 8 causes this sub-component to "highlight" (the faint bar 46 at the top of the left mouse 
button 18). Note that the input transducer 40 is still in the "tracking" state not a "mouse button 
down" state. A mouse down event (pen down over a mouse button) is depicted in figure 7. 
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When a "mouse button down" state occurs with the input transducer (e.g., with a pen, the pen 
tip is engaged by a tapping down of the pen onto the digitizer or tablet surface), the action of the 
tracking menu sub-component is executed. Tapping typically implies both the pressing down 
and lifting up of a pen on a surface. Here, by mouse button down when a stylus is involved is 
meant that the pen tip comes in contact with the surface. In this case, an event for the left 
mouse button 18 is generated, the bar 46 can be strongly highlighted, the pen down cursor 
position 40 can highlighted by a visible state change, such as the hot spot star shown in figure 
7 to indicate that a hotspot action is occurring at the tip of the arrow cursor 16. That is, any 
object, such as a menu item, at the tip of the arrow cursor or tracking symbol 16 is activated. 
[0045] The pen-mouse 1 0 of the present invention can not only provide the traditional 
simple mouse tracking symbol and single mouse button functions discussed above but can also 
provide more complicated functions as discussed below. 

[0046] Positioning the input transducer 40 over the body 50 of the mouse 1 0 can cause a 
pan tracking symbol 52 to appear as depicted in figure 8. A pen down event within the mouse 
body subcomponent 50 of the tracking menu 1 0 can cause the tracking symbol 40 to change to 
a more bold visual state as shown in figure 9. In this condition with the pan function activated, a 
mouse drag event (dragging the pen tip over the surface of the tablet PC or digitizer while the 
pen tip is touching the surface) within the mouse body 50 causes the tracking menu 10 to be 
dragged a distance and direction corresponding to the transducer 40 motion as illustrated .n 
figure 10. 

[0047] Positioning the input transducer 40 over a bottom tab 60 of the pen-mouse 10 can 
causes a bar tracking symbol 62 to appear (see figure 11) indicating the user is over this 
subcomponent. Once again, a mouse down event within the mouse bottom tab subcomponent 
60 of the tracking menu 10 causes the pen tracking symbol 40 to change to a more bold visual 
state (see figure 12) and invokes the subcomponent action. In this case, the subcomponent 
action, as illustrated in figure 1 3 is to invoke a set of marking menus 70. The behavior of the 
menus 70 matches that of the traditional marking menus. In figure 14, we see the user 
selecting "Alt" 72 which is a "check-box" item (i.e., remains enabled until the "Alt" item is 
explicitly selected again to toggle the state off). A pen-up event over the "Alt" menu item turns 
"Alt-lock" on and feedback is provided on the body of the mouse (the "A" character 74 of figure 
15) A pen-down on the bottom tab again brings up the marking menu 70 (see figure 16). 
Selecting Alt 72 again (see figure 17) turns off "Alt-lock" (see figure 18). Selecting "Middle" 76 
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from the marking menu 70 (see figure 19) can turn on a "Middle-lock" (see the bar 76 of figure 
20 indicating this state) that results in always generating a corresponding middle button mouse 
down/up event when any of the virtual pen-mouse buttons are engaged for a down/up action 
(see Figure 19). Figure 20 shows both "Middle-lock" and "Alt-lock" turned on. Hovering the pen 
40 over the left button 18 in this mode (as depicted in figure 22) followed by a pen-down on the 
pen-mouse 10 left mouse button 18 in this mode can cause left mouse button18 and middle 
mouse button events to be generated as depicted by the highlighted pen cursor symbol 40 of 
figure 23. This pair of events is focused on any object at which the arrow cursor 16 is pointing. 
That is, initiating an event that requires a two button "click" on a mouse can be accomplished 
using the pen-mouse 10 of the present invention. 

[0048] Figure 24 depicts a left mouse button drag event in progress. In this operation, the 
user has positioned the arrow cursor 16 on top of the "Folder" icon 90 and has engaged the left 
mouse button 18 of the pen-mouse 10 and begun dragging the pen 40 across the tablet while 
the tip is touching, thereby dragging the pen-mouse that drags the folder 90. 
[0049] In figure 25 the user has engaged the right mouse button by placing the tracker 
symbol 40 over the right mouse button component 22 of the pen-mouse 1 0 and performing a 
down event with the input transducer. This results in a pop-up menu 92 appearing, in this case 
to the left of the arrow cursor. Figure 26 shows dragging the pen downward (see the moved 
position of the highlighted pen tracking symbol as compared to figure 25) and this causes 
various menu items to highlight (in this case the "Reset Position" item 94) as one would expect. 
That is, the menu 10 can behave like a standard pop-up menu. 

[0050] The pen-mouse of the present invention need not use the arrow as the tracking 
symbol and can have a tracking menu boundary or edge that does not coincide with the visible 
edge of the mouse graphic. These alternatives are depicted in figure 27 where the pen-mouse 
tracking symbol 110 is a "hotspof type symbol and the tracking menu border 112 is a circle 
much larger that the graphic 114. 

[0051] The tracking symbol used with the pen-mouse, such as the arrow, can be positioned 
or oriented anywhere around the mouse graphic body. This choice in positioning can be 
designated by the user or managed by the system. For example, the system can move the 
arrow toward a display edge to allow easier selection of edge items. Examples of different 
positions for the symbol 120 are shown in figures 28a-28g and the utility or usefulness of each 
position includes: 28a - screen top edge objects, 28b - left handed users and top right screen 
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edge objects, 28c - right edge objects, 28d - torn right edge objects, 28e - bottom screen edge 
objects, 28f - bottom left edge objects, and 28g - left edge objects. 
[0052] The pen-mouse can have a number of different appearances, shapes or 
body/graphic designs. Figure 29 illustrates a pen-mouse 130 with a different shape and 
different functions including a wheel where the virtual wheel can be dragged up/down and the 
same actions as would occur when rotating a wheel are performed. Figure 30 depicts a pen- 
mouse 140 where only the buttons 142 are shown, where the tracking symbol 144 is a 
predetermined distance from the buttons 142 and the tracking border or boundary 146 has 
additional space on the right. Figure 31 illustrates a pen-mouse 150 with traditional L, M, R 
buttons and modifier keys. The tracking boundary 154 is depicted by a dashed line and 
coincides with the mouse graphic 156. This pen-mouse 150 includes a neck 158 attached to a 
base function region 60. The neck prevents the pen tracking symbol 162, a hot spot in this 
case, from accidentally triggering the functions of the base region 160. Figure 32 shows a pen- 
mouse 1 70 shaped like an arrow or a dragable cursor with LMR buttons and an action hot spot 
at the tip 172 of the arrow. The pen tracking symbol 172 is shown in the body. 
[0053] Figure 33 shows a pen-mouse 1 80 in which traditional changeable cursor states are 
shown. The tracking or action symbol 182 may also change as a function of the system state. 
The tracking symbol may also not change and rather only the system cursor changes which is 
attached to the tracking menu mouse body. 

[0054] Figure 34 depicts a pen-mouse 1 90 that includes an interior tracking wall 1 92 jutting 
from the tracking boundary that coincides with of the pen-mouse graphic. When the pen 
tracking cursor encounters this wall 192 the pen-mouse is moved similarly to the cursor 
encountering the exterior tracking boundary. Figure 35 illustrates a pen mouse 200 with a 
tracking boundary 202 having tracking wall 204 extending into the interior and that rotates the 
mouse graphic 206 when the pen cursor 208 encounters the wall 204. This allows the pen- 
mouse 200 to function like a virtual track ball. 

[0055] When the pen-mouse 1 0 of the present invention is displayed on a typical pen based 
computer display, the pen-mouse 1 0 is preferably displayed at a size similar to other tools of 
computer display as shown in figure 36. When the pen-mouse 10 is moved about on the 
display is preferably maintained on top of all objects in the display including menus and other 
persistent objects. 
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[0056] The hardware of the pen-mouse tracking menu system can be within a desktop PC 
230 (see figure 37) or within a handheld device, such as a tablet PC 240 (see figure 38) or a 
PDA, and includes an input transducer 360 the position of which is tracked by a computer 252 
that processes the transducer positions and creates the tracking menu display that is presented 
on a display 254 as shown in figure 39. 

[0057] The system also includes permanent or removable storage, such as magnetic and 
optical discs, RAM, ROM, etc. on which the process and data structures of the present invention 
can be stored and distributed. The processes can also be distributed via, for example, 
downloading over a network such as the Internet. 

[0058] The present invention has been described with respect to using a tablet type 
personal computer. The present invention is also suitable for other types of systems including 
large display formats (front and rear projection) both horizontal and vertical large displays, such 
as electronic whiteboards. Other input devices than a pen can also be used. For example, an 
audience could be viewing a large display presentation and if a user wants to manipulate the 
display from their seat, they could use a laser pointer type device. Here the pen tip down event 
would be generated by a dwell event or a secondary button on the laser pointer. The present 
invention has also been described with respect to a single input device being used with the 
system. The invention is also operable with two or more PenMice active at the same time, 
driven by two or more input devices. This allows two handed interaction techniques or 
collaborative multi-user applications. 

[0059] The many features and advantages of the invention are apparent from the detailed 
specification and, thus, it is intended by the appended claims to cover all such features and 
advantages of the invention that fall within the true spirit and scope of the invention. Further, 
since numerous modifications and changes will readily occur to those skilled in the art, it is not 
desired to limit the invention to the exact construction and operation illustrated and described, 
and accordingly all suitable modifications and equivalents may be resorted to, falling within the 
scope of the invention. 
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